Predicting the death by cancer rate for US counties

Simon Kennedy

Notebook 1 - Preprocessing

Problem definition

Predicting the death by cancer rate for US counties.

import sys

print(sys.version)

3.8.8 (default, Apr 13 2021, 15:08:03) [MSC v.1916 64 bit (AMD64)]

1. Loading the dataset

The original location of the file:

https://data.world/nrippner/ols-regression-challenge/file/cancer_reg.csv

Data Dictionary

TARGET_deathRate: Dependent variable. Mean per capita (100,000) cancer mortalities(a)

avgAnnCount: Mean number of reported cases of cancer diagnosed annually(a)

avgDeathsPerYear: Mean number of reported mortalities due to cancer(a)

incidenceRate: Mean per capita (100,000) cancer diagoses(a)

medianIncome: Median income per county (b)

popEst2015: Population of county (b)

povertyPercent: Percent of populace in poverty (b)

studyPerCap: Per capita number of cancer-related clinical trials per county (a)

binnedInc: Median income per capita binned by decile (b)

MedianAge: Median age of county residents (b)

MedianAgeMale: Median age of male county residents (b)

MedianAgeFemale: Median age of female county residents (b)

Geography: County name (b)

AvgHouseholdSize: Mean household size of county (b)

PercentMarried: Percent of county residents who are married (b)

PctNoHS18_24: Percent of county residents ages 18-24 highest education attained: less than high school (b)

PctHS18_24: Percent of county residents ages 18-24 highest education attained: high school diploma (b)

PctSomeCol18_24: Percent of county residents ages 18-24 highest education attained: some college (b)

PctBachDeg18_24: Percent of county residents ages 18-24 highest education attained: bachelor's degree (b)

PctHS25_Over: Percent of county residents ages 25 and over highest education attained: high school diploma (b)

PctBachDeg25_Over: Percent of county residents ages 25 and over highest education attained: bachelor's degree (b)

PctEmployed16_Over: Percent of county residents ages 16 and over employed (b)

PctUnemployed16_Over: Percent of county residents ages 16 and over unemployed (b)

PctPrivateCoverage: Percent of county residents with private health coverage (b)

PctPrivateCoverageAlone: Percent of county residents with private health coverage alone (no public assistance) (b)

PctEmpPrivCoverage: Percent of county residents with employee-provided private health coverage (b)

PctPublicCoverage: Percent of county residents with government-provided health coverage (b)

PctPubliceCoverageAlone: Percent of county residents with government-provided health coverage alone (b)

PctWhite: Percent of county residents who identify as White (b)

PctBlack: Percent of county residents who identify as Black (b)

PctAsian: Percent of county residents who identify as Asian (b)

PctOtherRace: Percent of county residents who identify in a category which is not White, Black, or Asian (b)

PctMarriedHouseholds: Percent of married households (b)

BirthRate: Number of live births relative to number of women in county (b)

(a): years 2010-2016

(b): 2013 Census Estimates

https://data.world/nrippner/ols-regression-challenge

import numpy as np
import pandas as pd

cancer_df = pd.DataFrame(pd.read_csv("C:\\Users\\skent\\OneDrive\\Documents\\1. GradDip Data Science UA\\5. Python B\\Assessments\\Ass 2\\Datasets\\cancer_reg.csv"))
cancer_df.head()                 

	avgAnnCount	avgDeathsPerYear	TARGET_deathRate	incidenceRate	medIncome	popEst2015	povertyPercent	studyPerCap	binnedInc	MedianAge	...	PctPrivateCoverageAlone	PctEmpPrivCoverage	PctPublicCoverage	PctPublicCoverageAlone	PctWhite	PctBlack	PctAsian	PctOtherRace	PctMarriedHouseholds	BirthRate
0	1397.0	469	164.9	489.8	61898	260131	11.2	499.748204	(61494.5, 125635]	39.3	...	NaN	41.6	32.9	14.0	81.780529	2.594728	4.821857	1.843479	52.856076	6.118831
1	173.0	70	161.3	411.6	48127	43269	18.6	23.111234	(48021.6, 51046.4]	33.0	...	53.8	43.6	31.1	15.3	89.228509	0.969102	2.246233	3.741352	45.372500	4.333096
2	102.0	50	174.7	349.7	49348	21026	14.6	47.560164	(48021.6, 51046.4]	45.0	...	43.5	34.9	42.1	21.1	90.922190	0.739673	0.465898	2.747358	54.444868	3.729488
3	427.0	202	194.8	430.4	44243	75882	17.1	342.637253	(42724.4, 45201]	42.8	...	40.3	35.0	45.3	25.0	91.744686	0.782626	1.161359	1.362643	51.021514	4.603841
4	57.0	26	144.4	350.1	49955	10321	12.5	0.000000	(48021.6, 51046.4]	48.3	...	43.9	35.1	44.0	22.7	94.104024	0.270192	0.665830	0.492135	54.027460	6.796657

5 rows × 34 columns

cancer_df.shape

(3047, 34)

Checking data types

cancer_df.dtypes

avgAnnCount                float64
avgDeathsPerYear             int64
TARGET_deathRate           float64
incidenceRate              float64
medIncome                    int64
popEst2015                   int64
povertyPercent             float64
studyPerCap                float64
binnedInc                   object
MedianAge                  float64
MedianAgeMale              float64
MedianAgeFemale            float64
Geography                   object
AvgHouseholdSize           float64
PercentMarried             float64
PctNoHS18_24               float64
PctHS18_24                 float64
PctSomeCol18_24            float64
PctBachDeg18_24            float64
PctHS25_Over               float64
PctBachDeg25_Over          float64
PctEmployed16_Over         float64
PctUnemployed16_Over       float64
PctPrivateCoverage         float64
PctPrivateCoverageAlone    float64
PctEmpPrivCoverage         float64
PctPublicCoverage          float64
PctPublicCoverageAlone     float64
PctWhite                   float64
PctBlack                   float64
PctAsian                   float64
PctOtherRace               float64
PctMarriedHouseholds       float64
BirthRate                  float64
dtype: object

2. Pre-processing the dataset

a. Cleaning

Creating a copy of cancer_df to make changes to

cancer_1 = cancer_df.copy()

All variables are the correct type

If they were not then the astype() function could be used. E.g. if I preferred the 'avgAnnCount' to be an integer rather than a float, I would use the code below:

cancer_1.avgAnnCount = cancer_1.avgAnnCount.astype( 'int64' )

The categorical variables will be converted to numerical before the modelling process as further cleaning is necessary on these varibles before they are ready for use.

Exploring the dataset

cancer_1.describe().transpose()

	count	mean	std	min	25%	50%	75%	max
avgAnnCount	3047.0	606.338544	1416.356223	6.000000	76.000000	171.000000	518.000000	3.815000e+04
avgDeathsPerYear	3047.0	185.965868	504.134286	3.000000	28.000000	61.000000	149.000000	1.401000e+04
TARGET_deathRate	3047.0	178.664063	27.751511	59.700000	161.200000	178.100000	195.200000	3.628000e+02
incidenceRate	3047.0	448.268586	54.560733	201.300000	420.300000	453.549422	480.850000	1.206900e+03
medIncome	3047.0	47063.281917	12040.090836	22640.000000	38882.500000	45207.000000	52492.000000	1.256350e+05
popEst2015	3047.0	102637.370528	329059.220504	827.000000	11684.000000	26643.000000	68671.000000	1.017029e+07
povertyPercent	3047.0	16.878175	6.409087	3.200000	12.150000	15.900000	20.400000	4.740000e+01
studyPerCap	3047.0	155.399415	529.628366	0.000000	0.000000	0.000000	83.650776	9.762309e+03
MedianAge	3047.0	45.272333	45.304480	22.300000	37.700000	41.000000	44.000000	6.240000e+02
MedianAgeMale	3047.0	39.570725	5.226017	22.400000	36.350000	39.600000	42.500000	6.470000e+01
MedianAgeFemale	3047.0	42.145323	5.292849	22.300000	39.100000	42.400000	45.300000	6.570000e+01
AvgHouseholdSize	3047.0	2.479662	0.429174	0.022100	2.370000	2.500000	2.630000	3.970000e+00
PercentMarried	3047.0	51.773679	6.896928	23.100000	47.750000	52.400000	56.400000	7.250000e+01
PctNoHS18_24	3047.0	18.224450	8.093064	0.000000	12.800000	17.100000	22.700000	6.410000e+01
PctHS18_24	3047.0	35.002068	9.069722	0.000000	29.200000	34.700000	40.700000	7.250000e+01
PctSomeCol18_24	762.0	40.977034	11.115805	7.100000	34.000000	40.400000	46.400000	7.900000e+01
PctBachDeg18_24	3047.0	6.158287	4.529059	0.000000	3.100000	5.400000	8.200000	5.180000e+01
PctHS25_Over	3047.0	34.804660	7.034924	7.500000	30.400000	35.300000	39.650000	5.480000e+01
PctBachDeg25_Over	3047.0	13.282015	5.394756	2.500000	9.400000	12.300000	16.100000	4.220000e+01
PctEmployed16_Over	2895.0	54.152642	8.315064	17.600000	48.600000	54.500000	60.300000	8.010000e+01
PctUnemployed16_Over	3047.0	7.852412	3.452371	0.400000	5.500000	7.600000	9.700000	2.940000e+01
PctPrivateCoverage	3047.0	64.354939	10.647057	22.300000	57.200000	65.100000	72.100000	9.230000e+01
PctPrivateCoverageAlone	2438.0	48.453774	10.083006	15.700000	41.000000	48.700000	55.600000	7.890000e+01
PctEmpPrivCoverage	3047.0	41.196324	9.447687	13.500000	34.500000	41.100000	47.700000	7.070000e+01
PctPublicCoverage	3047.0	36.252642	7.841741	11.200000	30.900000	36.300000	41.550000	6.510000e+01
PctPublicCoverageAlone	3047.0	19.240072	6.113041	2.600000	14.850000	18.800000	23.100000	4.660000e+01
PctWhite	3047.0	83.645286	16.380025	10.199155	77.296180	90.059774	95.451693	1.000000e+02
PctBlack	3047.0	9.107978	14.534538	0.000000	0.620675	2.247576	10.509732	8.594780e+01
PctAsian	3047.0	1.253965	2.610276	0.000000	0.254199	0.549812	1.221037	4.261942e+01
PctOtherRace	3047.0	1.983523	3.517710	0.000000	0.295172	0.826185	2.177960	4.193025e+01
PctMarriedHouseholds	3047.0	51.243872	6.572814	22.992490	47.763063	51.669941	55.395132	7.807540e+01
BirthRate	3047.0	5.640306	1.985816	0.000000	4.521419	5.381478	6.493677	2.132616e+01

Some variables have less than count = 3047, suggesting missing or na values.

• PctSomeCol18_24 = less than 3047 count
• PctEmployed16_Over = less than 3047 count
• PctPrivateCoverageAlone = less than 3047 count

There are some obvious potential outliers in variables:

• avgAnnCount.max = 38150
• avgDeathsPerYear.max = 14010
• studyPerCap.max = 9762.3
• MedianAge.max = 624
• BirthRate.max = 21.33

Checking for missing values

cancer_1.isnull().sum()

avgAnnCount                   0
avgDeathsPerYear              0
TARGET_deathRate              0
incidenceRate                 0
medIncome                     0
popEst2015                    0
povertyPercent                0
studyPerCap                   0
binnedInc                     0
MedianAge                     0
MedianAgeMale                 0
MedianAgeFemale               0
Geography                     0
AvgHouseholdSize              0
PercentMarried                0
PctNoHS18_24                  0
PctHS18_24                    0
PctSomeCol18_24            2285
PctBachDeg18_24               0
PctHS25_Over                  0
PctBachDeg25_Over             0
PctEmployed16_Over          152
PctUnemployed16_Over          0
PctPrivateCoverage            0
PctPrivateCoverageAlone     609
PctEmpPrivCoverage            0
PctPublicCoverage             0
PctPublicCoverageAlone        0
PctWhite                      0
PctBlack                      0
PctAsian                      0
PctOtherRace                  0
PctMarriedHouseholds          0
BirthRate                     0
dtype: int64

PctSomeCol18_24 has 2285 na values. This is 75% of the data. It is best ot remove this variable.

PctEmployed16_Over has 152 na values, which is only 5% of the data. The na data will be replaced with the column mean.

PctPrivateCoverageAlone has 609 na values, which is 20% of the data. The na data will be replaced with the column mean.

Dropping the variable 'PctSomeCol18_24' and replacing the na values with the variable mean in 'PctEmployed16_Over' and 'PctPrivateCoverageAlone'

cancer_1.drop(columns = ['PctSomeCol18_24'], inplace=True)

cancer_1.fillna(cancer_1.mean(), inplace = True)

cancer_1.isnull().sum()

avgAnnCount                0
avgDeathsPerYear           0
TARGET_deathRate           0
incidenceRate              0
medIncome                  0
popEst2015                 0
povertyPercent             0
studyPerCap                0
binnedInc                  0
MedianAge                  0
MedianAgeMale              0
MedianAgeFemale            0
Geography                  0
AvgHouseholdSize           0
PercentMarried             0
PctNoHS18_24               0
PctHS18_24                 0
PctBachDeg18_24            0
PctHS25_Over               0
PctBachDeg25_Over          0
PctEmployed16_Over         0
PctUnemployed16_Over       0
PctPrivateCoverage         0
PctPrivateCoverageAlone    0
PctEmpPrivCoverage         0
PctPublicCoverage          0
PctPublicCoverageAlone     0
PctWhite                   0
PctBlack                   0
PctAsian                   0
PctOtherRace               0
PctMarriedHouseholds       0
BirthRate                  0
dtype: int64

Comment:

Creating another copy of the dataframe

cancer_2 = cancer_1.copy()

Histogram plots of variables with potential outliers

'Average annual count' - mean number of reported cases of cancer diagnosed annually

import seaborn as sns
import matplotlib.pyplot as plt

sns.set(style="darkgrid")

fig = plt.figure(figsize=(10, 3))

sns.histplot(data=cancer_2, x="avgAnnCount",  color="purple")
plt.title("Full histogram of Average annual count of reported cases of cancer")
plt.show()

fig = plt.figure(figsize=(10, 6))

sns.histplot(data=cancer_2, x="avgAnnCount",  color="purple")
plt.ylim(bottom=0, top=20)
plt.title("Zoomed histogram of Average annual count of reported cases of cancer to highlight outliers")
plt.show()

There are 2 obvious outliers at approx. 25000 and 38150. Let's remove these values.

cancer_2 = cancer_2[(cancer_2.avgAnnCount < 15000)]
cancer_2.avgAnnCount.max()

14477.0

Mean deaths per year

fig = plt.figure(figsize=(10, 3))

sns.histplot(data=cancer_2, x="avgDeathsPerYear",  color="olive")
plt.title("Full histogram of Mean deaths per year")
plt.show()

fig = plt.figure(figsize=(10, 6))

sns.histplot(data=cancer_2, x="avgDeathsPerYear",  color="olive")
plt.ylim(bottom=0, top=20)
plt.title("Zoomed histogram of Mean deaths per year to highlight outliers")
plt.show()

There are 2 obvious outliers at approx. 9200 and 14010. Let's remove anything above 6000.

cancer_2 = cancer_2[(cancer_2.avgDeathsPerYear < 6000)]
cancer_2.avgDeathsPerYear.max()

5108

Per capita number of cancer - related clinical trials per county

fig = plt.figure(figsize=(10, 3))

sns.histplot(data=cancer_2, x="studyPerCap",  color="red")
plt.title("Full histogram of Studies per capita")
plt.show()


fig = plt.figure(figsize=(10, 6))

sns.histplot(data=cancer_2, x="studyPerCap",  color="red")
plt.ylim(bottom=0, top=20)
plt.title("Zoomed histogram of Studies per capita to highight outliers")
plt.show()

There are 3 obvious outliers at approx. 6800, 9500 and 9762. Let's remove any entries greater than 6000.

cancer_2 = cancer_2[(cancer_2.studyPerCap < 6000)]
cancer_2.studyPerCap.max()

4938.271605

Median Age

fig = plt.figure(figsize=(10, 3))

sns.histplot(data=cancer_2, x="MedianAge",  color="teal")
plt.title("Full histogram of Median Age")
plt.show()

fig = plt.figure(figsize=(10, 6))

sns.histplot(data=cancer_2, x="MedianAge",  color="teal")
plt.ylim(bottom=0, top=20)
plt.title("Zoomed histogram of Median Age to highlight outliers")
plt.show()

Obvious outliers appear in this variable. Lets filter out values > 100

cancer_2 = cancer_2[(cancer_2.MedianAge < 100)]
cancer_2.MedianAge.max()

65.3

Birth rate - Number of live births relative to number of women in county

fig = plt.figure(figsize=(10, 3))

sns.histplot(data=cancer_2, x="BirthRate",  color="orange")
plt.title("Full Histogram of Birth Rate")
plt.show()


fig = plt.figure(figsize=(10, 6))

sns.histplot(data=cancer_2, x="BirthRate",  color="orange")
plt.ylim(bottom=0, top=20)
plt.title("Zoomed Histogram of Birth rate to highlight outliers")
plt.show()

There is an obvious outlier at 21.36. Let's remove this.

cancer_2 = cancer_2[(cancer_2.BirthRate < 20)]
cancer_2.BirthRate.max()

18.55670103

creating another copy

cancer_3 = cancer_2.copy()

b. Dealing with categorical variables

Split 'Geography' column into 'County' and 'State' by delimiter

cancer_3[['County', 'State']] = cancer_3['Geography'].str.split(',', expand=True)
# display the new columns
cancer_3.iloc[:,30:].head()

	PctOtherRace	PctMarriedHouseholds	BirthRate	County	State
0	1.843479	52.856076	6.118831	Kitsap County	Washington
1	3.741352	45.372500	4.333096	Kittitas County	Washington
2	2.747358	54.444868	3.729488	Klickitat County	Washington
3	1.362643	51.021514	4.603841	Lewis County	Washington
4	0.492135	54.027460	6.796657	Lincoln County	Washington

Dropping 'Geography'column as it is not necessary. 'County' is simply an ID column so will be removed

cancer_3.drop(columns = ['Geography'], inplace=True)

cancer_3.drop(columns = ['County'], inplace=True)

# checking that the shape is as expected
cancer_3.shape

(3009, 33)

cancer_3.State.nunique()

51

# Import label encoder
from sklearn import preprocessing
  
label_encoder = preprocessing.LabelEncoder()
  
# Encode labels in column 'State'.
cancer_3['State']= label_encoder.fit_transform(cancer_3['State'])
  
cancer_3['State'].unique()

array([47, 48, 49, 27, 28, 29, 30, 31, 32, 46, 22, 23, 33, 34,  0,  3,  4,
       26, 42, 43, 18, 19, 20, 21, 44, 45,  5, 50, 24, 25, 16, 17,  6,  7,
        8,  9, 36, 37, 35, 38, 39, 40, 14, 15, 10, 11, 12, 13,  1,  2, 41])

cancer_3.State.nunique()

51

Re-formatting the 'binnedInc' elements to Levels 1 - 10 to make it easier to interpret

Level 1 being the lowest income group and Level 10 the highest income group.

The element will be purely numerical, i.e. integers to enable use in the modeling process.

cancer_3.binnedInc.unique()

array(['(61494.5, 125635]', '(48021.6, 51046.4]', '(42724.4, 45201]',
       '(51046.4, 54545.6]', '(37413.8, 40362.7]', '(40362.7, 42724.4]',
       '(54545.6, 61494.5]', '(34218.1, 37413.8]', '[22640, 34218.1]',
       '(45201, 48021.6]'], dtype=object)

cancer_3.binnedInc.replace('(61494.5, 125635]', 10, inplace = True)
cancer_3.binnedInc.replace('(54545.6, 61494.5]', 9, inplace = True)
cancer_3.binnedInc.replace('(51046.4, 54545.6]', 8, inplace = True)
cancer_3.binnedInc.replace('(48021.6, 51046.4]', 7, inplace = True)
cancer_3.binnedInc.replace('(45201, 48021.6]', 6, inplace = True)
cancer_3.binnedInc.replace('(42724.4, 45201]', 5, inplace = True)
cancer_3.binnedInc.replace('(40362.7, 42724.4]', 4, inplace = True)
cancer_3.binnedInc.replace('(37413.8, 40362.7]', 3, inplace = True)
cancer_3.binnedInc.replace('(34218.1, 37413.8]', 2, inplace = True)
cancer_3.binnedInc.replace('[22640, 34218.1]', 1, inplace = True)
cancer_3.binnedInc.unique()

array([10,  7,  5,  8,  3,  4,  9,  2,  1,  6], dtype=int64)

cancer_3.shape

(3009, 33)

c. Correlations

pair plot

import matplotlib.pyplot as plt
import seaborn as sns
sns.set()

sns.pairplot(cancer_3)

<seaborn.axisgrid.PairGrid at 0x1d900affe50>

This is too crowded to be very useful. Feature names are unreadable.

Correlation Heatmap

correlation_matrix = cancer_3.corr()

plt.figure(figsize=(12,10))
sns.heatmap(data=correlation_matrix, annot = True, fmt='.2f', linewidths=.5)

plt.show()

#Pearson's correlation 
corr = cancer_3.corr()

#Getting target_correlation and using the abs to unsigned the negative correlation 
corr_target = abs(corr['TARGET_deathRate'])
print('These are the highly correlated features\n',
      corr_target[corr_target>0.5])

These are the highly correlated features
 TARGET_deathRate    1.0
Name: TARGET_deathRate, dtype: float64

There are no highly correlated features

Further investigation into feature importance using the Wrapper Method Using Backward Elimination

# Using the statsmodels library 
import statsmodels.api as sm 
#We divide our data in training and testing dataset 
from sklearn.model_selection import train_test_split

X = cancer_3.drop(columns = ['TARGET_deathRate'])
y = cancer_3['TARGET_deathRate']

X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=42) #We define a seed to get the same split in the data 

###############
#OLS  
##############
#We added a constant from the features 
X_train1 =sm.add_constant(X)

#Fit the model 
model = sm.OLS(y,X_train1).fit()

#Getting the p-values and names 
print('p-values:')
ind=0
first = True
for i in model.pvalues:
    if first:
        print('const', i)
        first = False
    else: 
         print(cancer_3.columns[ind], i)
         ind = ind+1

p-values:
const 3.150864934999296e-25
avgAnnCount 0.000159213062791659
avgDeathsPerYear 1.4287456629989926e-06
TARGET_deathRate 4.832926529499207e-140
incidenceRate 0.0003152995352463348
medIncome 0.0003803179247702211
popEst2015 0.5168174985082609
povertyPercent 0.9111409326879498
studyPerCap 3.5361652578458685e-06
binnedInc 0.9462804524400575
MedianAge 0.40691054117609016
MedianAgeMale 0.7154587012855185
MedianAgeFemale 0.4756500050033693
AvgHouseholdSize 7.482767692137338e-14
PercentMarried 0.018548778690345716
PctNoHS18_24 1.2878238236995021e-05
PctHS18_24 0.1848383776271188
PctBachDeg18_24 3.824185812388407e-05
PctHS25_Over 5.591866802548286e-14
PctBachDeg25_Over 9.73723942709991e-07
PctEmployed16_Over 0.21388562516125728
PctUnemployed16_Over 0.004524579126554944
PctPrivateCoverage 0.5765433587579938
PctPrivateCoverageAlone 0.0013373280052767628
PctEmpPrivCoverage 0.13026040558204208
PctPublicCoverage 0.07697600128257788
PctPublicCoverageAlone 0.012933324116459456
PctWhite 0.31554254262371945
PctBlack 0.7404686035514334
PctAsian 6.340741085535792e-12
PctOtherRace 2.3252342437750765e-16
PctMarriedHouseholds 1.751441692844028e-05
BirthRate 0.032435768898269775

#Loop 
features = list(cancer_3.columns)
P_max=1
while(len(features)>0):
    X_train1 = cancer_3[features]
    X_train1 =sm.add_constant(X_train1)
    model = sm.OLS(y,X_train1).fit()
    #create a series with pvalues and names 
    series_pvalue = pd.Series(model.pvalues.values[1:], index = features)
    #Eliminated the feature with the biggest p_value
    p_max = max(series_pvalue)
    feature_with_p_max = series_pvalue.idxmax()
    if(p_max>0.05):
        features.remove(feature_with_p_max)
    else:
        break

print('Features with a p-value lower than 0.05 \n',features)

Features with a p-value lower than 0.005 
 ['TARGET_deathRate', 'popEst2015']

With only 1 variable have a p-value lower than 0.05 This does not inform the feature selection process any further.

Target correlations to inform which variables to drop due to collinearity.

corr['TARGET_deathRate']

avgAnnCount               -0.161466
avgDeathsPerYear          -0.098417
TARGET_deathRate           1.000000
incidenceRate              0.448593
medIncome                 -0.427573
popEst2015                -0.138447
povertyPercent             0.429486
studyPerCap               -0.015312
binnedInc                 -0.442996
MedianAge                 -0.004912
MedianAgeMale             -0.022346
MedianAgeFemale            0.012176
AvgHouseholdSize          -0.037944
PercentMarried            -0.267142
PctNoHS18_24               0.084910
PctHS18_24                 0.265034
PctBachDeg18_24           -0.289873
PctHS25_Over               0.405026
PctBachDeg25_Over         -0.485190
PctEmployed16_Over        -0.397491
PctUnemployed16_Over       0.379606
PctPrivateCoverage        -0.385234
PctPrivateCoverageAlone   -0.326254
PctEmpPrivCoverage        -0.264909
PctPublicCoverage          0.404315
PctPublicCoverageAlone     0.449242
PctWhite                  -0.179016
PctBlack                   0.259391
PctAsian                  -0.184982
PctOtherRace              -0.187702
PctMarriedHouseholds      -0.294223
BirthRate                 -0.086409
State                     -0.039808
Name: TARGET_deathRate, dtype: float64

The are several features that are highly correlated with each other (i.e. corr > 0.8). They are listed below with the correlation value with the TARGET_deathRate.

• avgAnnCount (-0.161) + avgDeathsPerYear (-0.0984) + popEst2015 (-0.138)
• medIncome (-0.428) + binnedInc (-0.443)
• MedianAge (-0.00491) + MedianAgeMale (-0.0223) + MedianAgeFemale (0.0122)
• PctPrivateCoverage (-0.385) + PctPrivateCoverageAlone (-0.326) + PctEmpPrivCoverage (-0.265)

For each group, the feature with the highest correlation to TARGET_deathRate will be kept and the others dropped, apart from the MedianAge group. I will keep both MedianAgeMale and MedianAgeFemale and drop Median Age.This was done to maintain an even number of genders and reduce potential bias.

Feature selection

Storing the target and the features that will feed the model

making another copy first

cancer_4 = cancer_3.copy()
cancer_4.shape

(3009, 33)

cancer_4.drop(columns = ['avgDeathsPerYear', 'popEst2015',
                            'medIncome', 'MedianAge',
                            'PctPrivateCoverageAlone','PctEmpPrivCoverage'], inplace=True)
cancer_4.shape

(3009, 27)

Re naming the dataset

cancer_clean = cancer_4.copy()
cancer_clean.head()

	avgAnnCount	TARGET_deathRate	incidenceRate	povertyPercent	studyPerCap	binnedInc	MedianAgeMale	MedianAgeFemale	AvgHouseholdSize	PercentMarried	...	PctPrivateCoverage	PctPublicCoverage	PctPublicCoverageAlone	PctWhite	PctBlack	PctAsian	PctOtherRace	PctMarriedHouseholds	BirthRate	State
0	1397.0	164.9	489.8	11.2	499.748204	10	36.9	41.7	2.54	52.5	...	75.1	32.9	14.0	81.780529	2.594728	4.821857	1.843479	52.856076	6.118831	47
1	173.0	161.3	411.6	18.6	23.111234	7	32.2	33.7	2.34	44.5	...	70.2	31.1	15.3	89.228509	0.969102	2.246233	3.741352	45.372500	4.333096	47
2	102.0	174.7	349.7	14.6	47.560164	7	44.0	45.8	2.62	54.2	...	63.7	42.1	21.1	90.922190	0.739673	0.465898	2.747358	54.444868	3.729488	47
3	427.0	194.8	430.4	17.1	342.637253	5	42.2	43.4	2.52	52.7	...	58.4	45.3	25.0	91.744686	0.782626	1.161359	1.362643	51.021514	4.603841	47
4	57.0	144.4	350.1	12.5	0.000000	7	47.8	48.9	2.34	57.8	...	61.6	44.0	22.7	94.104024	0.270192	0.665830	0.492135	54.027460	6.796657	47

5 rows × 27 columns

cancer_clean.shape

(3009, 27)

Saving the data with the to_csv function, for use in each of the different models

d. Normalisation

Normalisation will be done as part of the model pipeline in each notebook using MinMaxScaler( )

e. PCA

Data Segregation

X = cancer_clean.drop(columns = ['TARGET_deathRate'])
y = cancer_clean['TARGET_deathRate']

X.shape, y.shape

((3009, 26), (3009,))

from sklearn.preprocessing import MinMaxScaler
from sklearn.decomposition import PCA
from sklearn.pipeline import Pipeline
    
pipeline = Pipeline([('scaler', MinMaxScaler()),
                    ('pca', PCA())])
pipeline.fit(X,y)

Pipeline(steps=[('scaler', MinMaxScaler()), ('pca', PCA())])

pca = pipeline.named_steps.pca

plt.plot(np.cumsum(pca.explained_variance_ratio_))
plt.xlabel("Number of componets")
plt.ylabel("Cum Variance")

Text(0, 0.5, 'Cum Variance')

It looks like approximately 9 components explain at least 90% of the data. Lets check this with .

ve = pd.DataFrame(np.cumsum(pca.explained_variance_ratio_), columns = ['cumulative variance'], 
                  index = [1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26])
ve

	cumulative variance
1	0.387873
2	0.547387
3	0.680207
4	0.730034
5	0.777364
6	0.812891
7	0.841718
8	0.862557
9	0.881977
10	0.897255
11	0.911837
12	0.925079
13	0.937236
14	0.947429
15	0.956627
16	0.963953
17	0.970913
18	0.977202
19	0.981976
20	0.986243
21	0.989947
22	0.993218
23	0.995977
24	0.997643
25	0.999148
26	1.000000

Actually 11 components are required to explain at least 90% of the variance.

pca = PCA(n_components =11).fit(X)

#fit transform the 11 selected components
principalComponents = pca.fit_transform(X)

principalComponents.shape

(3009, 11)

New dataframe of 11 PC's

#Lets create the dataframe with 5 new components 
PC11_df = pd.DataFrame(data = principalComponents, columns = ['PC1','PC2','PC3','PC4','PC5','PC6', 'PC7', 'PC8', 'PC9', 'PC10', 'PC11'])
PC11_df.head()

	PC1	PC2	PC3	PC4	PC5	PC6	PC7	PC8	PC9	PC10	PC11
0	832.274962	313.546148	33.657837	-12.214108	10.698738	20.074393	-0.349950	3.997831	-3.125093	9.705851	-5.506573
1	-414.429848	-101.675206	-34.046802	-11.083636	20.573628	13.805643	21.197157	-4.148055	-10.468073	-1.504082	-1.371803
2	-484.561526	-74.401673	-96.093518	-7.794085	-3.682394	18.212923	2.445891	-5.847125	2.886886	8.086289	-3.055258
3	-144.810792	204.722249	-19.592955	-4.852360	-13.368746	22.599767	3.919350	0.002226	-1.134377	4.565621	-3.028927
4	-531.924237	-119.638489	-94.753719	-11.433868	-8.673093	19.826073	-4.799244	-6.655637	-5.343589	8.449074	-2.407916

print("Explained variance per principal component: \n", pca.explained_variance_ratio_)

Explained variance per principal component: 
 [8.72967813e-01 1.24101063e-01 1.99527243e-03 3.93958388e-04
 1.55171716e-04 1.31917515e-04 5.76903564e-05 4.34818781e-05
 3.98932180e-05 2.56582136e-05 1.82075848e-05]

print("Amount of information lost due to PCA:", (1-np.sum(pca.explained_variance_ratio_)))

Amount of information lost due to PCA: 6.987198570207465e-05

0.007% lost due to PCA

Establishing a baseline (6.a.i.)

Dummy regressor baseline

Splitting the data first and using a seed for consistency throughout the project

from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(X,y, random_state = 42)
# test_size is default 0.25 if train_size also none. i.e. train = 0.75, test = 0.25
# defining a seed to get the same split in the data

from sklearn.dummy import DummyRegressor
from sklearn.metrics import mean_squared_error

dummy_regr = DummyRegressor(strategy="mean") 
dummy_regr.fit(X_train, y_train)
dummy_baseline = dummy_regr.predict(X_test)

print(dummy_baseline[0:5])
print('dummy_baseline shape ', dummy_baseline.shape)

dummy_MSE = mean_squared_error(y_test, dummy_baseline)
dummy_RMSE = np.sqrt(dummy_MSE)
 
print("Dummy Mean square error: ", dummy_MSE)
print("Dummy RMSE: ", dummy_RMSE)

[178.3572695 178.3572695 178.3572695 178.3572695 178.3572695]
dummy_baseline shape  (753,)
Dummy Mean square error:  823.3448438000677
Dummy RMSE:  28.693986195718217